{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### http://blog.csdn.net/marsjhao/article/details/72900646"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 本文使用的数据集是CIFAR-10，这是一个经典的数据集，包含了60000张32*32的彩色图像，其中训练集50000张，测试集10000张，如同其名字，CIFAR-10数据集一共标注为10类，每一类6000张图片，这10类分别是airplane、automobile、bird、cat、deer、dog、frog、horse、ship和truck。类别之间没有重叠，也不会一张图片中出现两类物体，其另一个数据集CIFAR-100则标注了100类\n",
    "\n",
    "![http://img.blog.csdn.net/20170607155257721?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvbWFyc2poYW8=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast][url]\n",
    "[url]:http://img.blog.csdn.net/20170607155257721?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvbWFyc2poYW8=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 本节的神经网络中，我们采用卷积神经网络CNN，还是用了一些新的技巧：\n",
    "1. 对权重weights进行L2正则化；\n",
    "2. 对图片数据进行数据增强，即对图片进行翻转、随机剪切等操作，制造了更多的样本；\n",
    "3. 在每个卷积-池化层后使用了LRN层（局部相应归一化层），增强了模型的泛化能力。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 说明\n",
    "> 使用tf.nn.lrn()函数，即对结果进行LRN处理。LRN层（局部响应归一化层）模仿了生物神经系统的“侧抑制”机制，对局部神经元的活动创建竞争机制，使得其中响应比较大的值变得相对更大，并抑制其他反馈较小的神经元，增强了模型的泛化能力。LRN对ReLU这种没有上限边界的激活函数比较试用，不适合于Sigmoid这种有固定边界并且能抑制过大值的激活函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 数据增强DataAugmentation\n",
    "\n",
    "数据增强在我们的训练中作用很大，包括了随机的水平翻转、随机剪切图片、设置随机的亮度和对比度以及对数据进行标准化等操作。它可以给单幅图增加多个副本，提高图片的利用率，防止对某一个图片结构的学习过拟合。如果神经网络能够克服这些噪声并准确识别，那么其泛化性能必然会很好。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 神经网络结构\n",
    "![http://img.blog.csdn.net/20170607155356099?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvbWFyc2poYW8=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast][url]\n",
    "[url]:http://img.blog.csdn.net/20170607155356099?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvbWFyc2poYW8=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf  \n",
    "import numpy as np  \n",
    "import time \n",
    "import cifar10, cifar10_input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 要使用cifar10, cifar10_input，需要下载model, 并进入相应文件夹内创建python文件\n",
    "> git clone https://github.com/tensorflow/models.git    \n",
    "> cd models/tutorials/image/cifar10\n",
    "\n",
    "- 参考: \n",
    "[http://blog.csdn.net/u012526003/article/details/61078726][url]\n",
    "[url]: http://blog.csdn.net/u012526003/article/details/61078726"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "max_steps = 1000\n",
    "batch_size = 128\n",
    "data_dir = \"../cifar-10-batches-bin/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def variable_with_weight_loss(shape, stddev, wl):\n",
    "    var = tf.Variable(tf.truncated_normal(shape, stddev=stddev))\n",
    "    if wl is not None:\n",
    "        weight_loss = tf.multiply(tf.nn.l2_loss(var), wl, name=\"weight_loss\")\n",
    "        tf.add_to_collection(\"losses\", weight_loss)\n",
    "    return var"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cifar10.maybe_download_and_extract()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.\n"
     ]
    }
   ],
   "source": [
    "images_train, labels_train = cifar10_input.distorted_inputs(data_dir=data_dir, batch_size = batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_test, labels_test = cifar10_input.inputs(eval_data=True, data_dir=data_dir, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "image_holder = tf.placeholder(tf.float32, [batch_size, 24, 24, 3])\n",
    "label_holder = tf.placeholder(tf.float32, [batch_size])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weight1 = variable_with_weight_loss([5,5,3,64], stddev=5e-2, wl=0.0)\n",
    "kernel1 = tf.nn.conv2d(image_holder, weight1, strides=[1,1,1,1], padding=\"SAME\")\n",
    "bias1 = tf.Variable(tf.constant(0.0, shape=[64]))\n",
    "conv1 = tf.nn.relu(tf.nn.bias_add(kernel1, bias1))\n",
    "pool1 = tf.nn.max_pool(conv1, ksize=[1,3, 3,1], strides=[1,2,2,1], padding=\"SAME\")\n",
    "norm1 = tf.nn.lrn(pool1, 4, bias=1.0, alpha=0.001 / 9.0, beta=0.75)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weight2 = variable_with_weight_loss([5, 5, 64, 64], stddev=5e-2, wl=0.0)  \n",
    "kernel2 = tf.nn.conv2d(norm1, weight2, strides=[1, 1, 1, 1], padding='SAME')  \n",
    "bias2 = tf.Variable(tf.constant(0.1, shape=[64]))  \n",
    "conv2 = tf.nn.relu(tf.nn.bias_add(kernel2, bias2))  \n",
    "norm2 = tf.nn.lrn(conv2, 4, bias=1.0, alpha=0.001 / 9.0, beta=0.75)  \n",
    "pool2 = tf.nn.max_pool(norm2, ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1], padding='SAME')  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reshape = tf.reshape(pool2, [batch_size, -1]) # 将每个样本reshape为一维向量  \n",
    "dim = reshape.get_shape()[1].value # 取每个样本的长度  \n",
    "weight3 = variable_with_weight_loss([dim, 384], stddev=0.04, wl=0.004)  \n",
    "bias3 = tf.Variable(tf.constant(0.1, shape=[384]))  \n",
    "local3 = tf.nn.relu(tf.matmul(reshape, weight3) + bias3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weight4 = variable_with_weight_loss([384, 192], stddev=0.04, wl=0.004)  \n",
    "bias4 = tf.Variable(tf.constant(0.1, shape=[192]))  \n",
    "local4 = tf.nn.relu(tf.matmul(local3, weight4) + bias4)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weight5 = variable_with_weight_loss([192, 10], stddev=1 / 192.0, wl=0.0)  \n",
    "bias5 = tf.Variable(tf.constant(0.0, shape=[10]))  \n",
    "logits = tf.matmul(local4, weight5) + bias5  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 定义损失函数loss  \n",
    "def loss(logits, labels):  \n",
    "    labels = tf.cast(labels, tf.int64)  \n",
    "    cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(  \n",
    "            logits=logits, labels=labels, name='cross_entropy_per_example')  \n",
    "    cross_entropy_mean = tf.reduce_mean(cross_entropy, name='cross_entropy')  \n",
    "    tf.add_to_collection('losses', cross_entropy_mean)  \n",
    "    return tf.add_n(tf.get_collection('losses'), name='total_loss')  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'Tensor' object is not callable",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-15-9bde42c1af2b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlogits\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel_holder\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# 定义loss\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mtrain_op\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mAdamOptimizer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1e-3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mminimize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# 定义优化器\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mtop_k_op\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0min_top_k\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlogits\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfloat64\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlabel_holder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfloat64\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: 'Tensor' object is not callable"
     ]
    }
   ],
   "source": [
    "loss = loss(logits, label_holder) # 定义loss  \n",
    "train_op = tf.train.AdamOptimizer(1e-3).minimize(loss) # 定义优化器  \n",
    "top_k_op = tf.nn.in_top_k(tf.cast(logits, tf.float64), tf.cast(label_holder, tf.float64), 1)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Thread(QueueRunnerThread-input_producer-input_producer/input_producer_EnqueueMany, started daemon 140563089168128)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140563005306624)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562996913920)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562988521216)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562980128512)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562971735808)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562963343104)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562954950400)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562803980032)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562795587328)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562787194624)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562778801920)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562770409216)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562762016512)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562753623808)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562535544576)>,\n",
       " <Thread(QueueRunnerThread-shuffle_batch/random_shuffle_queue-shuffle_batch/random_shuffle_queue_enqueue, started daemon 140562527151872)>,\n",
       " <Thread(QueueRunnerThread-input/input_producer-input/input_producer/input_producer_EnqueueMany, started daemon 140562518759168)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140562510366464)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140562501973760)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140562493581056)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140562485188352)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561327585024)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561319192320)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561310799616)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561302406912)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561294014208)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561285621504)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561277228800)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561193367296)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561184974592)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561176581888)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561168189184)>,\n",
       " <Thread(QueueRunnerThread-batch/fifo_queue-batch/fifo_queue_enqueue, started daemon 140561159796480)>]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sess = tf.InteractiveSession()\n",
    "tf.global_variables_initializer().run()\n",
    "\n",
    "tf.train.start_queue_runners()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 迭代训练  \n",
    "for step in range(max_steps):  \n",
    "    start_time = time.time()  \n",
    "    image_batch, label_batch = sess.run([images_train, labels_train]) # 获取训练数据  \n",
    "    _, loss_value = sess.run([train_op, loss],  \n",
    "                             feed_dict={image_holder: image_batch,  \n",
    "                                        label_holder: label_batch})  \n",
    "    duration = time.time() - start_time # 计算每次迭代需要的时间  \n",
    "    if step % 10 == 0:  \n",
    "        examples_per_sec = batch_size / duration # 每秒处理的样本数  \n",
    "        sec_per_batch = float(duration) # 每批需要的时间  \n",
    "        format_str = ('step %d, loss=%.2f (%.1f examples/sec; %.3f sec/batch)')  \n",
    "        print(format_str % (step, loss_value, examples_per_sec, sec_per_batch))  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在测试集上测评准确率  \n",
    "num_examples = 10000  \n",
    "import math  \n",
    "num_iter = int(math.ceil(num_examples / batch_size))  \n",
    "true_count = 0  \n",
    "total_sample_count = num_iter * batch_size  \n",
    "step = 0  \n",
    "while step < num_iter:  \n",
    "    image_batch, label_batch = sess.run([images_test, labels_test])  \n",
    "    predictions = sess.run([top_k_op],  \n",
    "                           feed_dict={image_holder: image_batch,  \n",
    "                                      label_holder: label_batch})  \n",
    "    true_count += np.sum(predictions)  \n",
    "    step += 1  \n",
    "  \n",
    "precision = true_count / total_sample_count  \n",
    "print('precision @ 1 =%.3f' % precision) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
